Efficient Query Optimization for Distributed Join in Database Federation

Database federation is one approach to data integration, in which a middleware, called mediator, provides uniform access to a number of heterogeneous data sources. For the mediator, two key components are query rewriter and query optimizer. In this thesis, we focus on the query optimizer part, particularly, on cost-based query optimization for distributed joins over database federation. One important observation in query optimization over distributed database system is that run-time conditions (namely available buffer size, CPU utilization in machine and network environment) can significantly affect the execution cost of a query plan. However, in existing database federation systems, very few studies have addressed runtime conditions. It is a challenging problem, because usually the mediator is not able to know the run-time conditions of remote sites and considering run-time conditions will bring about extra complexity to the optimizer. This thesis proposes the Cluster-and-Conquer algorithm for query optimization over database federation while efficiently considering run-time conditions. I firstly propose to view the whole federation as a clustered system, by grouping data sources based on network infrastructure or enterprise boundaries; and then provide each cluster of data sources with its own cluster mediator. The query optimization is divided into two procedures: the global mediator decides inter-cluster operations, and cluster mediators handle the sub queries within the cluster with run-time condition consideration. This algorithm has three-fold benefits. Firstly, the run-time conditions of machines are now available for cluster mediator, because the communication within a cluster is timeefficient. Secondly, each cluster mediator can deal with its own sub query concurrently, so the complexity of processing query plan is decreased. Thirdly, the algorithm outperforms other related approaches in terms of “cost of costing”, because it removes unnecessary inter-cluster operations in the early stage of query plan selection. I have implemented a prototype data federation system with Cluster-and-Conquer algorithm. The experimental results showed the capabilities and efficiency of our algorithm and described the target scenarios where the algorithm performs better than other related approaches.